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PREFACE 



This report describes the results of a collaborative study conducted from 1984 to 
1986 to examine the implementation of competency-based adult education in California. The 
study combined the resources of two projects and the advice and assistance of many others : 
who are acknowledged below. 

The objective of the study was to determine the effects of California's competency- 
based adult education (CBAE) mandate on the functioning of adult basic education programs 
throughout the state. The CBAE mandate, initiated in 1982, required the state's P.L. Section 
306-funded adult education agencies to utilize a competency-based approach in delivering 
adult basic education services. The implementation of a competency-based approach was 
expected io result in a systematic and comprehensive outcome based delivery system for 
adult basic education in the state. 

This study was undertaken to investigate two questions: 

1) How have Section 306 agency personnel Interpreted the CBAE 
mandate? 

2) What have been the effects of the CBAE mandate on agency 
maragement, classroom functioning, and student outcomes? 

The findings and conclusions of !his report suggest several ways in which the CBAE mandate 
has resulted in an improved adult basic education system, as well as strategies for further 
strengthening of this system. 

Appreciation is extended to Claude Hansen, Manager of Adult Education Program 
Services, Unit California State Department of Education and to Jane Zinner, director of the 
Dissemination Network for Adult Education, who assisted in the design of the study, 
participated in study meetings, and reviewed drafts of the report. Others who provided 
advice on the study design included: Annie Herda, University of San Francisco; Carolyn 
Fowle, California Department of Education; John Martois, Los Angeles County Office of 
Education. 

Finally, thanks are given to the administrators, teachers, counselors, and students 
from the Section 306 adult education agencies that were visited as part of the study. This 
access was critical to the study's success. It is for them and their colleagues in adult 
education agencies across the state that this report is intended. 

The study was conducted under the auspices of the CBAE Staff Development Project, 
San Francisco State University, directed by John Tibbetts and David Hemphill, and the 
CASAS Project, San Diego Community College District Foundation, directed by Patricia 
Rickard. Both of these are California Department of Education-funded projects under federa! 
P.L. 91-230, Section 310. Special thanks are also due to Sylvia Ramirez, coordinator of 310 
mini-grants, for her work in coordinating the production and publicatior: of this report. The 
views and conclusions in this report are those of the authors and do not represent those of 
the California Department of Education or the federal government. 
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I. INTRODUCTION 



Competency-based adult education (CBAE) has had a rich history in California. For 
over ten years, elements of a competency-based process have existed in adult education 
programs throughout the state. In 1982 the practice of CBAE in California became more 
focused when the State Department of Education's Adult Education Field Services Unit 
mandated the use of a competency-based approach in the provision of basic education services 
to adults. That is, all agencies that wished to receive Section 306 fundings*, under the 
provisions of the Adult Education Act, had to structure their adult basic education programs 
to include the elements of a CBAE process. In the mandate, CBAE was defined as a 
"performance-based process leading to demonstrated mastery of basic and life skills 
necessary for the individual to function proficiently in society." (National Center for 
Educational Statistics, 1982, p.80.) 



Rationale for the CBAE Mandate 

The State Department of Education's decision to require the use of a competency-based 
approach was based on its prior experience funding CBAE projects within California, as well 
as increasing national attention on the application of competency-based processes to adult 
basic education. While adult education agencies had implemented some elements of a 
competency-based process, most programs were fragmented in their approach to service 
delivery. The implementation of the CBAE mandate was viewed as a mechanism for 
facilitating program coordination and for increasing the quality of the basic education 
services being delivered to adults. While agencies were expected to structure their programs 
to include key program components--for example, specification of competencies, procedures 
for placement, coordinated instruction and assessment, use of appropriate materials, and 
processes for reporting student performance results-flexibility was given in the selection of 
specific competencies and program content. 

In presenting a framework for organizing adult basic education programs, the State 
Department of Education sought to establish better procedures for program accountability, as 
well as a common reference for communication and sharing that would result in overall 
improvement of adult basic education in California. 



Purpose of the Study 

The CBAE mandate had a three-year time frame in which Section 306 programs, 
English as a Second Language (ESL), Vocational English as a Second Language (VESL), and Adult 
Basic Education (ABE/Basic Subjects), were to become competency-based. As the three 
years drew to a close, the need to assess the progress of the mandate became apparent to the 
Adult Education Field Services Unit. Thus, in August 1984, a study was initiated to determine 
the effects of the CBAE mandate on the functioning of Section 306 programs. 



*Monies allocated to the California Department of Education from the US Department of Education, under the 
provisions of the Adult Education Act, P.L 91-230, Section 306-hereafter referred to as Section 306. 
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In preparation for the study, an evaluation study team was convened in August, 1984, 
to determine the evaluation questions and types of data that would be required to assess 
adequately the effects of the CBAE mandate. It was determined that existing quantitative 
data, collected under the auspices of the Section 310** projects, wouid be combined with 
new structured interview data, to examine the impact of the CBAE mandate on the 
functioning of Section 306-funded adult basic education programs,. The study was undertaken 
to answer two research questions: 



1. How have Section 306-funded agency personnel Interpreted 
the CBAE mandate? 



What changes have occurred at the agency level, classroom 
level and student level that may be associated with the use 
of a competency-based approach? 



Viewed as a collaborative process combining internal project knowledge with 
external expertise about evaluation, the study was begun in late 1984. The study team met 
a number of times to give guidance to the data collection analysis and interpretation phases 
of the study, and to formulate recommendations about the future of CBAE in California. 

This report describes the results of the first year of the study- 1984-85. 
Presented in Chapter II is a discussion of California's experience in implementing 
competency-based adult education, and the factors considered in the establishment of the 
mandate. Chapter III describes the major projects that were developed to support the CBAE 
implementation process. Chapter IV contains a review of the methods of the study and 
Chapter V discusses findings. Presented in Chapter IV are 1) the conclusions about the 
effects of the CBAE mandate on adult education program functioning and 2) policy 
recommendations for the future of CBAE in California. 




"Monies allocated under the provisions of the Adult Education Act P.L 91-230, Section 310 -- hereafter referred 
to as Section 310. _ 
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II. DEVELOPMENT OF CBAE IN CALIFORNIA 



History of CBAE Implementation 

California's involvement with competency-based education can be traced at least to 
June 1974, when San Francisco State University, in conjunction with the Far West 
Laboratory for Educational Research and Development in San Francisco, sponsored the first 
state-wide CBAE Conference in San Diego. During the ensuing ten years, the California 
Department of Education's Field Services Unit increasingly supported the development of 
CBAE by funding new projects with the Section 309/310 monies provided under the Adult 
Education Act. These efforts resulted in the establishment of a diversity of programs, 
generally funded from one to three years. 

During the late 1970's projects were funded to improve curricula, to provide staff 
development, and to identify skills needed to enter the job market -e.g. the Adult 
Competency Education Project (ACE) in San Mateo; the California Adult Competency 
Education Project (CACE) at San Francisco State University; the Competency-Based 
Livability Skills Project (CLASS) at the Clovis Adult School; and the integrated Competency- 
Based Bilingual Vocational English as a Second Language Project (ICB-VESL) at Chinatown 
Resources Development Center in San Francisco. Some of these projects were disseminated 
nationally, such as the CLASS Project, which was funded as an exemplary program in the 
U.S. Department of Education's National Diffusion Network. As well, materials developed in 
the Class and ICB-VESL Projects were distributed nationally through commercial publishers. 

Also, during this time, the emphasis in California expanded to program improvement 
at the adult high-school level. Two major efforts undertaken were: 1) the development of 
the Competency-Based Activities Packets (CAPS) in the Los Angeles Unified School District, 
and 2) the California Competency-Based High School Diploma Project (CALCOMP), a 
consortium of seven adult education agencies that was coordinated by the Center for Adult 
Education, San Francisco State University. 

Parallel to the efforts improving high school programs was a new emphasis on 
dissemination. From 1976 to 1980, the Information, Collection, Evaluation, and 
Dissemination System (ICDES) was implemented by the San Diego Community College 
District, Continuing Education Division. Repositories, primarily housed in county office 
facilities, were established throughout the state as a way of providing access to adult 
education materials. 

As these innovative projects completed their funding cycles, agencies throughout the 
state began to initiate the implementation of a competency-based process, such as: 1) 
student needs assessment, 2) testing of competency attainment, and \ 3) increased focus 
on a life skills curriculum. A few agencies integrated basic (academic) and life skills 
application content in their programming. For most, however, the approach to CBAH was 
fragmented and sporadic. 




Development of the State Plan 



Spring of 1982 marked the beginning of a new thrust in CBAE in California. In the 
process of developing California's state plan for the expenditure of federal funds under the 
provisions of the Adult Education Act (California Department of Education, 1982) the Adult 
Education Field Services Unit Staff identified a number of deficiencies in the content and 
quality of adult basic education programs throughout the state. These deficiencies were: 

Excessive class size and instructor-student ratios; 

Curriculum procedures and content that were not keeping pace 

with contemporary trends and technology; 

Little or no provision for staff development; 

Little or no provision for comprehensive program evaluation; and 

Little or no provision for counseling and guidance services. 

In examining these deficiencies, it was determined that one approach for correcting 
these deficiencies would be the development of a competency-based system for delivering 
adult education services. Competency-based curriculum and staff development projects had 
previously been implemented in California, and a new competency-based assessment 
process-California Adult Student Assessment System (CASAS)-was near completion. It 
was thought that a competency-based system that incorporated a management framework, as 
well as processes for monitoring student progress and program completion, would provide 
the necessary structure for improving the quality of adult basic education services in 
California, as well as a mechanism for increased program accountability. 

Establishment of the Mandate 

The submission of a new state plan for using federal adult education monies provided 
an opportunity to stipulate new procedures for distributing fiscal resources. As well, the 
state staff, in consultation with adult education practitioners and project directors 
throughout the state, developed a framework for facilitating the coordinated delivery of adult 
basic education services and for sharing effective educational practices among adult 
education agencies. 

The result was a state mandate for recipients of Section 306 funds. The mandate 
stated that: 



"Funds will be distributed only to those agencies which can 
assure quality programming for the target population identified 
In this plan." (The California State Plan-1982 Submission, pp. 
19-20). in order to receive Section 306 funding, local adult 
education /agencies were requested to make provisions for the 
following: 

* Class sizes that do not exceed a certificated employee- 
to-student ratio of 30-1 on the average; 
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* Curriculum design and content that is based upon the 
philosophy, process and procedures of competency- 
based learning; 

* A student assessment system which is competency-based; 

* A program of ongoing staff development for all certificated 
staff Involved with the federal program; and 

* Counseling and guidance services to assure that all students 
are properly placed in the appropriate competency level, 
and that their progress is monitored with appropriate 
assessment instruments 



The creation of a mandate for competency-based adult education in California was a 
significant step toward the development of a comprehensive adult basic education delivery 
system. The key requirements of the mandate-that is, provision of appropriate class size, 
participation In staff development activities, use of competency-based curricula and 
assessment procedures, and provision Jor counseling and guidance services-were 
considered critical for improving the quality cf adult basic education. However, the mandate 
alone was not thought to be sufficient for accomplishing the goal of an Improved adult 
education system. In addition, a support system was needed to assist local adult education 
agencies in implementing the mandate. Section 310 monies available under the Adult 
Education Act provided the resources to develop coordinated support for the areas addressed 
in the mandate-assessment, staff development, and dissemination. 

Thus, '.he Field Services Unit Staff made the unprecedented decision to focus the 
distribution of Section 310 monies to support projects in the three areas noted above. This 
decision, along with the competency-based adult education mandate, represented the 
establishment of a major state policy initiative in adult ^asic education in this country. 



Summary 

The development of competency-based adult education in California progressed from 
the implementation of discrete curriculum and staff development projects by local agencies 
and state-funded projects to the establishment of a comprehensive delivery system for adult 
basic education. Two key policy decisions facilitated this development: 1) the creation o' \ 
mandate in 1982 requiring that all Section 306 adult education agencies utilize a competency- 
based approach in delivering services to adult clients, and 2) the focused distribution of 
Section 310 funds for projects in competency-based assessment, staff development, and 
dissemination, in order to support the implementation of the mandate. 

Other support mechanisms for assisting adult education agencies in competency-based 
program improvement were also developed. The variety of supports provided to adult 
education agencies for implementing the mandate are described in Chapter III. 



ERLC 



5 12 



III. SUPPORTS TO CBAE IMPLEMENTATION 



A major goal of California's CBAE mandate was to provide adult basic education 
agencies with common program standards for providing services to students. In order to 
accomplish this goal, it was recognized that a strong support system would have to be 
established to assist agencies in implementing the CBAE mandate. Thus, assessment, 
dissemination, and staff development projects were funded with Section 310 monies to 
provide the major segment of this support. There was also a need to create mechanisms 
through which adult education agencies could build internal support systems. It was clear 
that if the effects of the mandate's implementation were to persist, the building of both 
inter- and intra-agency capacities to facilitate change was critical. 



State Supports 

The building of the capacity for change at the agency level was facilitated by the State 
Department of Education through various means. Major assistance to agencies was provided 
through the activities of the Section 310 projects-California Adult Student Assessment 
System (CASAS), the Dissemination Network of Adult Educators (DNAE), and the CBAE Staff 
Development Project. The State Department of Education's investment in these projects 
represented a serious effort toward providing comprehensive support to statewide change. 
Each project focused on a different segment of the CBAE change process, and each was 
intended to support the work of the others. 

CASAS was designed to enable adult educators *o develop and evaluate a life skills 
curriculum linking instruction and assessment. The project is a comprehensive educational 
assessment system that provides adult education agencies with effective assessment 
materials and processes for all levels of adult basic education, English-as-a-Second Language 
(ESL), Vocational ESL (VESL) and high school completion programs. The project's goals are 
achieved through a cooperative planning consortium of 40 agencies in California as well as 50 
other states. 

DNAE was funded with the primary responsibility of locating, screening and making 
available for adoption exemplary programs for adults, thereby functioning as a clearinghouse 
for teachers and administrators. DNAE assists local agencies in identifying exemplary 
instructional programs and/or projects and identifies and disseminates information regarding 
copyrighted materials relevant to the field of adult education. 

The CBAE Staff Development Project was funded to assist agencies in the design and 
implementation of a competency-based adult education approach for Adult Basic Education, 
English-as-a-Second Language and VESL Programs. The project relies upon a consortium of 
experts from across the state, and it assists local agencies in developing CBAE 
implementation teams of management, guidance and instruction personnel. The project helps 
agencies to establish competency-based programs through self-assessment (ISAM) staff 
development mini-grants, workshops ,and support materials. 
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Another form of support was provided by S^iie Department of Education field 
consultants based in the Adult Education Field Services Unit, who were asked to assist the 
CBAE implementation process within each of their geographical areas of responsibility. The 
field consultants were encouraged to use the compliance review visit as an opportunity to 
support the progress of CBAE implementation. 

Another significant form of implementation support took the form of financial 
assistance, initiated at the beginning of 1983, the second year of the mandate. The State 
Department of Education awarded grants of $2,500--using Section 306 funds-to each 
agency (regardless of size) to assist in implementing the various components of the CBAE 
process. Staff development mini-grants of $600 also were made available by the CBAE Staff 
Development Project to agencies to further implement CBAE. 



Agency Supports 

Research on the educational change process has pointed to the importance of 
developing cooperative relationships among and between staff at a!! agency levels. In 
designing the implementation of the CBAE mandate, it was recognized thai adult education 
agency personnel involved in all aspects of agency service delivery-administration, 
instruction and guidance would need to work together to accomplish a common set of agency 
goals. It was thought that through the development of agency teams, a foundation for 
organizational change could be established. The involvement of all levels of personnel was 
considered critical to the success of the implementation process, particularly that of the 
agency administrator. These agency teams, combined with technical assistance provided by 
the state support system described previously, were seen as key elements for initiating and 
supporting the change process. As previous studies on dissemination and utilization in 
education have shown, the involvement of external facilitators as well as that of key 
personnel is essential if the desired organizational change is to be long-term. 



Agency Teams 

Since the establishment of an agency team was considered a primary part of the CBAE 
implementation process, much time and attention was given to organizing and guiding its 
development. Through the efforts of the CBAE Staff Development and CASAS projects, 
assistance was given to Section 306 agencies regarding the role and responsibilities of 
agency team members. In some agencies, this was an initial attempt to encourage 
cooperation among personnel-for others, this provided an opportunity for strengthening 
existing working relationships. 

Agencies were asked to form teams that consisted of: ABE, ESl, and VESL teachers; a 
management representative; and a guidance person-if one was employed by the agency. 
Thus, the teams generally consisted of from two to five persons, depending on the size of the 
agency. The teams, considered as the "cnange agents" for the CBAE implementation process, 
were given a number of activities to undertake in support of the adoption of a CBAE approach. 
These activities were focused on implementing the CBAE Program Framework and Program 
Processes listed on the next page: 
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CBAE PROGRAM FRAMEWORK 

Provide a Statement of Philosophy that reflects a CBAE approach; 

* Establish program purposes and timelines for completion of the CBAE 
implementation process; 

Conduct community, staff, and student needs assessments to determine 
local priorities; and 

Identify competencies that meet the student population and community 
needs, and design curriculum that addresses these competencies. 



CBAE PROGRAM PROCESS 

1. Assessment System 

* Develop individual student profiles. 

* Establish a placement process that includes asssessment 

of both basic and life skills and focuses on student goals and needs; 

* Establish a system for monitoring student and group progress, 
including a recordkeeping system; and 

* Establish criteria and implement assessment procedures for student 
movement to the next level or for certification upon exiting the 
program. 



2. Guidance System 



Identify the guidance roles needed in orientation to program, 
placement, monitoring, counseling, and certification of student 
achievement; 

Provide career guidance resources to support instruction and 
clarification of student goals and goal attainment; 

Maintain an open entry/open exit system; and 

Provide guidance that enables culturally, diverse students to move 
through, exit and change program levels. 



3. Instructional System 

Conduct student needs assessment at the classroom level; 
* Develop CBAE course outlines; 
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Provide instruction that integrates basic skills with life skills; 

Identify and use appropriate instructional materials for life roles; 

Conduct pre/post assessment for each instructional unit; 

Use a variety of instructional strategies appropriate to diverse 
student needs and learning styles; and 

Provide opportunities for application of skills to real life. 

4. Management System 

Design a program to enable students to attain competencies; 

Develop, implement and maintain a staff development program; and 

Implement program evaluation including appropriate feedback to 
teachers and students. 



The CBAE Program Framework and Program Processes provided a blueprint for 
Section 306 agencies to follow in implementing the CBAE mandate. The activities specified in 
both were considered critical for supporting the adoption of a CBAE approach and for 
achieving long-term program improvement. 



Summary 

In instituting the CBAE mandate, the State Department of Education recognized that 
the change required in the adult basic education agencies would have to be facilitated by a 
variety of resources-i.e., .human, material, psychological, and financial. Thus, a 
comprehensive support system was developed to provide the necessary technical assistance 
to agencies. Through this directed approach to building capacity at the local level, it was 
hoped that Section 306 agency personnel would move their adult basic education programs 
toward becoming competency-based, and subsequently, would provide more consistent 
services to their adult students. 

This study was undertaken to determine what some of the effects of this effort have 
been. The data collection methods used in this study to determine in what ways the tenets of 
the CBAE mandate have been realized are described in Chapter IV. 
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IV. METHODS 



In order to investigate the effects of the implementation of a competency-based 
process, a multiple-method approach to data collection was undertaken. This approach 
utilized a combination of quantitative and qualitative data collection strategies. These two 
strategies were selected so thai critical aspects of the CBAE implementation process could b 
captured in as many descriptive and analytic ways as possible. 

The data collection process was designed to accomplish two goals. First, there was a 
desire to obtain a better understanding of the changes in the agencies, instructors, students, 
and classes that have occurred as a result of using competency-based approaches. The 
second goal was to obtain a clear understanding of the variety of factors influencing the 
Section 306 agencies' capacities to implement the CBAE mandate. Of particular interest were 
the ways in which agency personnel have come to interpret the tenets of the mandate, 
particularly with regard to the operational aspects of using competency-based program 
components. 

A combination of existing and new data was used to address the study questions. 
Existing data, which had been collected with instruments designed under the auspices of the 
Section 310 projects, were utilized to provide both demographic and impact data to describe 
the agencies, and to examine the effects of the implementation of the mandate on changes in 
organization, personnel, and student achievement. The collection of new data-in the form of 
interviews-was undertaken in order to gain an understanding of how the CBAE mandate had 
been interpreted and operationalized at the agency level. 

The choice of an approach that included the collection of 1) quantitative data about the 
results of using a competency-based system and 2) interview data about the CBAE 
implementation process provided a vast array of information from which conclusions could be 
drawn. The use of this approach also increased the likelihood that unintended effects of the 
CBAE implementation process could be identified. Finally, the combination of quantitative and 
interview data provided a comprehensive basis for formulating recommendations about the 
refinement of the CBAE implementation process. 



Sample 

Personnel and ABE/ESL students from Section 306 adult education agencies, 
implementing CBAE programs during 1982-1985, were the primary subjects of the study. 
Quantitative data obtained from personnel and ABE/ESL students in these agencies provided 
baseline information about the effects of using a competency-based system. To provide a 
clearer understanding of the implications of implementing CBAE, perceptual data gathered 
through structured interviews from a sample of personnel and ABE/ESL students in 11 of 
these agencies was used to gain an in-depth analysis of the utility of the CBAE process. In 
addition, programs at these 11 agencies were examined to determine the effects C3AE 
implementation had on agency organization, student achievement, and classroom activities. 
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The sample of Hagencies was selected according to variation on three 
dimensions: 1) agency size (large, medium and small), 2) geographical location (north, 
south, rural, urban) and 3) level of CBAE implementation (high, medium, low). An agency's 
level of CBAE implementation was determined by 1) its progress toward implementing the 
key elements of a CBAE program-as assessed by a panel of experts who had visited the 
eleven agencies, and 2) its rating on the Institutional Self-Assessment Measure (ISAM). Of 
the11 agencies studied, four were designated to be "high implementing", three "medium 
implementing", and four "low implementing". 

Participation in the study was voluntary and respondents were given assurances of 
their anonymity in reporting the data. In addition, formal exit interviews were conducted 
with agency personnel to provide information about impressions gained concerning the CBAE 
implementation process. 



Quantitative Data 

In order to understand the extent to which the CBAE mandate had been implemented, 
as well as the effects of the implementation process on student growth, classroom changes, 
and agency organization, instrumentation that had boen previously generated through the 
activities of the CASAS and Staff Development projects were analyzed. These instruments 
and the procedures used for collecting and analyzing the data are described below: 

Institutional Self-Assessment Measure (ISAM). The ISAM is an instrument that is 
used to assess the degree of CBAE implementation for each of the Section 306 agencies. Its 
design was based upon the minimum program elements for CBAE implementation that were 
identified in the California Adult Ba*>ic Education State Plan. Items and scoring criteria were 
reviewed for validity by a state-wide staff development consortium, and were refined for 
field-testing with 25 agencies by June 1983. Interrater reliability of 95% agreement 
among three independent raters, rating four different agencies, was obtained. 

The instrument included 89 items using a five-point behaviorally-anchored rating 
scale. The measure yields a total CBAE implementation profile score, as well as sub-scores 
for management, instruction, and guidance strands. The data for this study were based on 
29 key items that represent the minimum program elements which would constitute CBAE 
implementation. The data for a two-year period, 1983-84 and 1984-85, were examined for 
this study. 

The number and percent of agencies reporting implementation scores on a scale of 1-5 
were used for the study. For analysis purposes, "low implementing" was defined as scale 
ratings of from 1-3, while "high implementing" was defined as scale ratings of 4-5. 

CASAS Life Skills Survey Achievement Test The Survey Achievement Test is a 
three-level achievement test designed through item response theory methods to measure 
student progress across the state in beginning through advanced levels of ABE and ESL. As a 
requirement for receiving Section 306 funds, agencies are required to administer the test to 
a representative sample of their student population. The test assesses functional life skills 
literacy in consumer economics, community resources, health, occupational knowledge, and 
government and law. Four equivalent test forms were constructed for each program level 
and the content and range of item difficulty were matched in designing the equivalent forms. 
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As well, an achievement scale was developed for each test form. This single parameter 
Rasch scaling procedure permits measurement of student progress on an equal interval scale, 
independent of norm groups. The test also provides content information about student 
mastery, since every item has a scaled standardized difficulty level and is referenced to a 
specific life skills competency statement. Internal consistency reliability of the test items 
has been determined through extensive field testing. 

The results of tests given during 1982-85 were analyzed by comparing the average 
of the mean cognitive gain after approximately 100 hours of instruction of three "high 
implementing 11 agencies to the average of the means of three "low implementing" agencies. 
These six agencies were drawn from the 11 agencies that were the subject of the study. 
Further, the mean cognitive gain was compared between the "high" and "low" implementing 
agencies by program and level. 

ABE Class Questionnaire. The ABE Class Questionnaire was designed to collect class 
level variables, including class size, class hours per week, class setting, amount of class 
instruction on different curriculum areas, student goals, and learning levels. The 
questionnaire is administered in conjunction with the post testing of the CASAS Life Skills 
Survey Achievement Test. 

The results of the ABE Questionnaire, administered during 1982-85 in Section 306 
agencies, were analyzed by comparing the means of the percent of class time teaching life 
skills and attrition rate. Comparisons were made between the "high" and "low 1 implementing 
agencies, as previously defined. These items were then further compared by program and 
level. 

ABE 1 Report. The ABE 1 Report is prepared by agencies receiving Section 30* funds 
from the California State Department of Education. Two types of data are submitted in Jhis 
yearly report: 1) the total enrollment data for ESL and ABE classes, including numbers of 
teachers, enrollees, attendance hours, and instructional hours, and 2) the first week in 
March census estimating enrollments, by program level, ethnic group, and language group. 

Data reported in the documents submitted by all of the Section 306 agencies during the 
1982-85 CBAE implementation period were analyzed for 1) differences between services 
provided to ABE and ESL students and 2) enrollment trends. 

Assessment Survey. The Assessment Survey was developed to evaluate the uses and 
impact of the CASAS Survey Achievement pre and post tests. The instrument contains seven 
questions that are designed to collect information about the possible uses of the CASAS Life 
Skills Survey Achievement Test. Responses are collected through the use of a modified 
Likert scale for each item and space is provided for comments. 

The 286 responses from surveys sent to all Section 306 agencies during 1983-84 
were analyzed for purposes of this study. Cross break analysis of the Likert scale 
responses was applied using chi square to test inference for generalizability. The responses 
to the open-ended questions were coded for trends. 




Instructor Survey. The Instructor Survey was designed to document, in a systematic 
way, the changes that were being reported at the classroom level by the CASAS Consortium 
as they implemented CBAE/CASAS. The evaluation committee compiled a list of changes that 
CASAS Consortium members reported during the summer institute sessions. The instrument 
was developed and field-tested by a committee made up of CASAS Consortium members 
representing administration, guidance and instructional personnel and an external evaluation 
team. The following topics are assessed in the survey: 

* " Teaching modes or methods of presentation; 

Choice of new materials and use of old materials; 

* Frequency of testing/assessment; 
Focus of instruction, academic/life skills; 
Student interaction and enthusiasm; 
Attitude toward testing/assessment; 

* Use of community resources; 

* Communication with other instructors on program issues; and 

* Use of classroom aides. 



Instructors are asked to complete the survey based on their use of the CASAS 
system. The degree of change is rated for each of the variables-that is, topics-measured 
by the survey, on a scale of: 1) not at all or very little, 2) somewhat, or 3) significantly 
or a great deal. If the instructors report moderate to significant changes, they are asked to 
describe the change in more detail. 

For the purposes of this study, results were analyzed in terms of the percent of 
instructors reporting moderate to significant change. A total of 172 Instructor Surveys, 
from adult education agencies who are members of the CASAS Consortium, were included in 
the analysis. 



CASAS JDRP (Joint Dissemination Review Panel) Submission Data. CASAS JDRP 
Submission Data for assessing program effects on students' retention in adult education 
programs are included in this report. A non-equivalent control group design was used wiih 
intact classrooms being randomly assigned to treatment and control conditions. The 
classrooms involved in this study were ESL classrooms from two different sites in a fairly* 
large urban and suburban district. A total of 28 classrooms from these two sites 
participated in the study; within each site, 14 ESL classrooms received random assignment 
to treatment or contral conditions. Thus, the design was modified into a 2x2 factorial to 
account for an important element of external validity, namely, location (site 1 vs. site 2). 
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The measure used to assess student retention rate was program enrollment records. 
Specifically, the classroom test rosters for the state mandated CASAS Life Skills Survey 
Achievement Test were used to verify student enrollment on a pre and post test basis. For 
the purposes of this study, enrollment information was collected from all participating 
classrooms by the ABE/ESL program coordinator for the district. Pretest enrollment data 
were collected at the beginning of the Fall semester in 1983 and following 100 hours of 
instruction. Once the post test data were collected, all results were sent directly to an 
external evaluator for analysis. 

The analysis of enrollment results included both descriptive and inferential 
procedures. The total and average enrollments were competed for both sites previous to and 
following 100 hours of instruction. The percent of student retained in CASAS and non- 
CASAS classrooms also was computed. Two-way analyses of variance were run to compare 
average enrollment by site and treatment condition previous to and following 100 hours of 
instruction. 



Interview Data 

For the purposes of this study, interview protocols containing questions designed with 
an open-ended response format were developed to gather information about interpretations 
respondents had of the CBAE process and its effects. 

The eleven agencies visited were representative in terms of geographical location and 
CBAE implementalion level (previously described). A sub-sample of personnel and students 
was selected at each site for the collection of the interview data. The total sub-sample size 
for all agencies was 146. These included: 19 administrators, 49 teachers, nine counselors, 
five aides, and 64 students. A pilot test of the interview protocols was conducted in a 
twelfth agency, in which four administrators, six teachers, two counselors, two aides, and 
six students were interviewed.) 

In addition to the Section 306 agency personnel and ABE/ESL students, five State 
Education Department field consultants and five department leaders who have been involved 
in the implementation of the CBAE mandate were interviewed. Descriptions of the 
information gathered in the interviews, conducted during 1984-85, as well as the data 
collection and analysis procedures, are discussed below. 

Section 306 Agency Interviews. Interviews were conducted with a selection of 
administrators, teachers, students, counselors, and aides in each of the 11 agencies in the 
sample. The protocol questions for agency personnel were designed to elicit the 
interviewees 1 interpretations of the CBAE process and their perceptions of the effects of the 
implementation of the key elements of CBAE, as designated in the mandate. Separate 
protocols were created for each type of personnel. Topics covered in these interviews 
included: 

personal definition of CBAE (all personnel); 
shift in job responsibilities (all personnel); 
planning process (administrators, teachers); 
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* identification of competencies (teachers); 
student placement procedures (all personnel, students); 

* orientation to program (students); 
instructional strategies (Jeachers, students); 
use of materials (teachers, students) 
effective use of life skills approach (teachers); 
student assessment procedures (teachers, counselors/aides); 
participation in staff development (all personnel); 
program changes (all personnel); 
overall effects of CBAE mandate (administrators); 
program improvements (all personnel, students). 

In addition to administrators, teachers, counselors, and aides, a number of students 
were interviewed in each agency. The interviews with students were designed to obtain 
information about their prior experience attending adult education classes, as well as 
perceptions about their current educational oxperience. The average length of agency 
personnel interviews, conducted at the agency site, was one hour. Student interviews were 
approximately one-half hour. To assure accurate student responses, students with limited- 
English proficiency were interviewed in their native language. 

State Department of Education Interviews. A component of the data collection process 
was the gathering of information from the State Department of Education personnel who have 
been involved both in the design and implementation of the CBAE mandate. These individuals 
have been responsible for monitoring the implementation of the Section 306 programs and for 
facilitating the provision of services by the Section 310 projects. 

An interview protocol was designed and pilot-tested. Interviews, which averaged an 
hour in length, were conducted with the State Department of Education personnel at their 
offices. The major topics covered during the interviews were the following: 

operational definition of CBAE mandate; 

field consultants 1 role in facilitating the mandate; 

CBAE implementation process; 

major eifects of mandate; 

* recommendations for change. 

16 
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In addition, State Department of Education personnel were asked whether the 
implementation process and resulting system change have followed the direction that was 
originally intended in the mandate. 



Procedures for Analysis of Interview Data 

Two adult education field researchers, familiar with the CBAE implementation 
process, conducted all interviews for the two groups of individuals studied: 1) agency 
personnel and students, and 2) State Department of Education personnel. These field 
researchers also were responsible for the coding of the interview data, along with two other 
members of the project team. 

The interview data were pre-coded using a framework that was developed based on a 
sample of the initial interviews. A series- of categorical responses was established for each 
open-ended question. Provisions also were made for the inclusion of additional response 
categories. Inter-rater reliability was established by having study team members code the 
same questions and then compare results. .■ . 

Frequency distributions of selected protocol questions were tabulated for each 
category of interviewees-i.e., administrators, teachers, counselors/aides, and students. 
The interview responses of personnel and students from "high" and "low" implementing 
agencies, as originally designated in the selection of the sub-sample of agencies, were 
compared, ommonalities and differences in perceptions were identified among responses. 



Summary 

The combination of quantitative and interview measures described provided a broad 
database from which the effects of CBAE implementation could be determined. These 
measures permitted an examination of these effects both in terms of organization and 
individual outcomes. Presented in Chapter V are the findings from the analysis of these 
measures. 
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CHAPTER V: FINDINGS OF THE STUDY 



This chapter discusses the findings of the study, which are divided into two sections: 
A, The "Key Elements" of CBAE Implementation, and B. Other Results of the 
Study. Each finding is stated, followed by supporting data drawn from both quantitative and 
interview data sources. 



A. The "Key Elements" of CBAE Implementation 



The nine findings in this section pertain to what are agreed to be "key elements" of 
CBAE implementation in California. All nine of these findings are supported by multiple data 
sources. 



1. Identified Competencies 

Approximately one-half of the state's section 306 agencies 
have gone through the process of identifying competencies, 



Supporting Data. The percentage of agencies using competency lists to guide 
program organization has remained constant. A comparison of ISAM scores from 1983-84 
(n=64 agencies) through 1984-85 (n=56 agencies) indicates no .significant positive shift in 
percentage of agencies with competency lists. In 1984-85, 50% of agency managers (n=28) 
reported that they had developed competency lists, while 39% of instructors representing 
17 agencies reported similarly. 

Five of 11 agencies visited for interviews had identified competencies and were 
using a competency list to guide the program. Five other agencies used a textbook to identify 
competencies, and one had not identified program competencies. 

Of those programs in the interview sample that were using competency lists, a few 
had done limited local needs assessment, but most had adopted the CASAS competency list or 
another agency's list, without apparent adaptation to local program needs. The agencies 
without competency lists appeared to have implemented some CBAE program components, 
independent of a set of competencies. 



2. Prepare Student Profiles 

From the management and guidance perspective, the number of 
agencies that have developed student profiles has increased. 
From the instructor perspective, there has been less progress. 
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Supporting Data. A comparison of ISAM scores from 1983-84 to 1984-85 
reveals an increase in the percentage of agencies reporting the preparation of student 
profiles (a student profile includer ^formation beyond that which is normally collected in 
class registration, including previous educational experience, student goals, and previous 
work experience). This finding holds true for managers (shift from 28% to 58%), 
instructors (25% to 39%), and guidance personnel (21% to 56%), 

In three of 11 agencies in the interview sample, student profiles were being used 
as part of an intake process to gather information such as students' previous education, work 
experience, and future goals. In one of these agencies, it was reported that data gathered on 
student goals had been used to restructure the ESL program into three tracks: academic, 
vocational, and survival English, 

Although the percentage of the agencies in the interview sample using student profiles 
was smaller than the percentage of student profile use reported in the ISAM analysis, 
instructor? interviewed reported that gathering and maintaining such data was a useful form 
of record-keeping, 

3. Monitor, Record, and Report Student Progress 

There has been a substantial increase in the use of the CASAS 
Survey Achievement Test to monitor group progress in agencies 
throughout the state, and in using the results for program and 
Instructional planning. Instructors indicated less progress than 
did management and guidance personnel. Where test results 
are scored by the agency for instructors, and returned quickly 
In an understandable format, then instructors generally have 
been able to make good use of the data. 



Supporting Data, This key element describes the use and interpretation of the 
CASAS Survey Achievement Tests, An overall increase in the use of CASAS Survey 
Achievement Tests was reported between the school years 1983-84 and 1984-85, While the 
number of agencies participating in Section 306 funding has remained relatively stable over a 
three-year period (1982-83, 1983-84, and 1984-85), the number of agencies, classrooms, 
and students participating in pre-and post-testing has increased, as noted in Graphs 1a & 
1b below: 

GRAPHS 1a & 1b 
Extent of CASAS Pre-Post Testing Over a Three-Year Period 
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1982-83 1983-84 1984-85 1982-83 1983-84 1084-85 
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These graphs indicate that agencies were both assessing more students and returning more 
valid test scores, although there was no change in the state's mandate that agencies needed, 
at a minimum, to test a representative sample of their classes using the CASAS instrument. 
When patterns of valid test scores are examined further, 74% of the "high implementing" 
agencies 1 pre-post test scores for the overall program are va'ld, compared to 57% for "low 
implementing*'. 

CASAS Instructor Survey results also support the finding of a general increase in the 
monitoring, reporting, and recording of student progress among agencies implementing a 
competency-based assessment system. Thirty-six percent of instructors (n=168) reported 
moderate to significant change in frequency of assossment, and 34% (n=163) reported 
moderate to significant change in student attitudes toward testing and assessment. CASAS 
Instructor Survey results also indicated that 56% of the total sample regarded CASAS pre/ 
post test results as extremely useful for '^proving instruction. 

Furthermore, a comparison of ISAM scores from 1983-84 (n=63) to 1984-85 (n=56) 
indicates an increase in the percentage of agencies reporting acceptable levels of monitoring, 
recording, and reporting student progress. The reported changes are most significant in the 
responses of guidance personnel (14% to 61%) and managers (24% to 47%), while results 
are marginally significant for instructors (21% to 34%). 

In five of 11 agencies where interviews were conducted, it was reported that 
CASAS Survey Achievement Test results were scored for instructors, anc? returned quickly 
in an understandable format. In these cases, instructors saw the testing process as useful. 
Seven of the 11 agencies visited for interviews were members of the CASAS Consortium. 
Four of these seven agencies were processing and returning test results in the manner 
described above, with similar positive results. 



4. Document Student Competency Attainment 

Although change In documenting individual student competency 
attainment has been statistically significant in Instruction 
and guldance y the level of implementation still remains low. 
Interview data showed that in ESL, due to class size and 
method of Instruction, Individual competency attainment 
tracking remains problematic. In ABE, the technical process 
of recording individual student progress is well-handled, but 
it is oriented toward basic skills, and not life skills. 



Supporting Data. No other key element of CBAE implementation seemed to 
generate as much controversy from respondents in the 11 sites visited to collect interview 
data as did recording student competency attainment. Attainment of competencies by 
individual ESL students was being documented in only two of the 11 agencies visited. 

A comparison of ISAM scores from 1983-84 (n=64 agencies) to 1984-85 (n=56 
agencies) shows a marginal increase in the percentage of agencies reporting the 
documentation of student competency attainment. In 1984-85, 27% of the managers 
responding reported that documentation of student competency attainment was taking place 
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in their agencies. While both instructors and guidance personnel reported some increase in 
documentation of competency attainment, a two-year average of percentages across 
guidance, instruction, and management would estimate implementation at no more than 25% 
of agencies statewide. 

In three of the four agencies designated in the interview sample as "high 
implementing", the majority of instructors commented in interviews that keeping records of 
individual student competency attainment - particularly in ESL classes - was "not helpful," 
"not attainable," or "not feasible." There was persistent concern that it took bo much time 
away from classroom instruction. 

A comment frequently expressed by the 38 ESL instructors interviewed across all 
11 sites visited was that record-keeping required instructors to "play too many roles." 
While six of thel 1 agencies had aides that assisted with placing students and organizing 
curriculum, no agency had classroom aides that performed a student competency record- 
keeping function. 

Some ESL instructors questioned the appropriateness of documenting individual 
r *udent competency attainment in ESL classes. According to one instructor, whose comment 
v.as illustrative of a commonly-held opinion, "Recording of individual competencies is not as 
important if there isn't an individualized program of instruction. Language acquisition 
depends on a communicative approach." 

In interviews with 11 ABE instructors, however, all reported that they recorded 
individual student progress, but the focus was on basic skills, not life skills. 

One possible explanation for the disparity in ESL and ABE instructor attitudes about 
documenting competency attainment may be found in the differences between class size for 
ESL and ABE populations statewide. Class size data reported on the ABE-1 Survey over a 
three-year period (1982- 83, 1983-84, and 1984-85) show consistently larger average 
class size in ESL than in ABE. 




GRAPH 2 





According to this graph, the process of managing larger numbers of students may make the 
documentation process difficult. 
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In interviews with ESL instructors and managers, the development of uniform unit 
tests and level exit tests seemed to be one new and emerging way to address the issue of 
documenting individual student competency attainment. 

5. Place Student Into Programs 

Most agencies established a structured system for placing 
students into programs relatively early in the CBAE 
implementation process, and the level of use of such 
systems remains at a high level. All 11 agencies visited 
to collect interview data had some organized form of 
placement system in place. 

Of the nine"key elements 1 * of CBAE implementation included in the mandate, interview 
data suggest that placement of students is the element that has been most consistently and 
effectively implemented. All eleven agencies studied had some established system for placing 
students into classes at the appropriate level. 

As Graph 3 below indicates, a comparison of ISAM scores from 1983-84 to 1984-85 
shows an increase in the percentage of agencies reporting implementation of competency- 
based placement systems. 



GRAPH 3 



Implementatic , of Competency-Based Placement Systems 




Instructors 
Guidance 
Managers 





* (n refers to number of agencies) 

Instruments used to assess ESL students for placement purposes often were developed 
locally some from the CASAS Item Bank, while others were taken from other agencies. 
Placement systems were implemented through a combination of ingenious arrangements, 
frequently including the use of aides or clerical staff. 

Some respondents noted that prior to the CBAE mandate, ESL classes in their agencies 
were not separated into levels at all, and thus were not seen to require a placement process. 
Rather, there were three or four multi-level classes being offered side by side in the same 
site, at the same time. In these agencies, it was reported that CBAE implementation was 
associated with the development of separate levels of classes, and with placement systems 
to enable referral of students to appropriate levels. 
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In the five agencies visited where there were ABE programs, placement and intake 
interviews were performed by counselors. In ESL programs, however, placement was 
generally conducted by instructors or aides. There have been attempts to use existing 
counselors to place ESL students, but it has not always been successful. As an illustration of 
this, one resource instructor commented, "ESL instructors don't have the confidence in 
counselors to assess language development, or possibly ESL instructors don't want to give up 
control." Another agency also had experimented unsuccessfully with using a counselor for 
placement of ESL students. Reasons suggested for this lack of success were, "too many 
students were placed incorrectly," and, "just the sheer number of students made it 
impossible for the counselor." 

One agency did have success in using a counselor to place ESL students, but this was 
because a counselor was hired who had been specifically trained to counsel the ESL 
population. 



6. Counsel Students for Movement and Change in Program 

This component is more frequently implemented by counselors 
working with ABE students and high school diploma students. 
There was little evidence of counseling focus on ESL students. 
Instructors appear to be filling this role to some extent with 
ESL students. 



Supporting Data. A comparison of ISAM scores from 1983-84 (n=64) to 1984-85 
(n=55) indicates an increase in the percentage of agencies reporting the use of activities 
designed to counsel students for movement and change in program. Instructors reported a 
17% increase (from 41% to 58%) in the delivery of such services. The most significant 
change was noted by guidance personnel, 75% of whom reported in 1984-85 (n=18) that such 
counseling programs were in place, 31% increase over 1983-84. The relatively small 
sample of 18 agencies with guidance personnel reporting, however, would suggest that 
overall, there are rather limited guidance services available to ABE/ESL students. 

Interview data, moreover, suggested problems in several areas that might be 
classified as "guidance" functions, including: orientation of students, maintaining profiles of 
individual student backgrounds and goals, enabling students to move through and between 
programs, and accessing community resources. As previously noted under the key element of 
"placement," ESL students consistently received less support from counselors than did ABE 
students, where both programs were offered within the same agency. 

ESL students in all 11 agencies visited reported that they had heard about the 
program from a friend or relative. In some cases, they mentioned receiving fliers from the 
adult education agency. Most students reported that on their first day of ESL classes, they 
had gone to an office, filled out an application, and had been assessed through some kind of 
placement procedure. They then were referred directly to a classroom instructor. They 
were usually not told about available educational opportunities other than ESL classes. As an 
extreme example of this, one student reported that he wanted to take high school diploma 
classes, but didn't know where they were offered (they were provided at a center directly 
across the street). 
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In ESL programs, instructors were generally responsible for deciding when to move 
students from one level to another. Instructors were also most often responsible for 
providing information about community services to students. 

In smaller agencies, instructors felt that key guidance functions were handled 
effectively through informal communication. In medium and large agencies instructors were 
less comfortable with how effectively guidance services were provided to ESL students. 

In contrast with the limited counselor support provided to ESL students, a higher 
level of counselor support was provided to ABE and high school diploma program students. Of 
the12 ABE students interviewed in the11 agencies studied, all students reported having been 
interviewed by counselors when they entered the program. Counselors, in turn, consistently 
reported that most of their time was devoted to working with ABE and high school diploma 
program students, although ABE and high school diploma program student populations 
combined were consistently smaller than ESL program student populations. The only 
interactions reported by counselors with ESL students on a regular basis came about through 
visits to advanced ESL classes, where counselors encouraged ESL students to continue their 
education in ABE and high school diploma classes. 

In summary, the ISAM data reported initially in this section seems to suggest a 
substantial increase in the delivery of guidance services. However, because of the small 
sample of agencies reporting this ISAM data, and because of convincing interview data 
regarding the lack of guidance services for ESL students, it may be concluded that there is a 
continuing problem in the area of guidance for ESL students that remains to be addressed. 



7. Use Appropriate Instructional Materials 

There has been an increase in the use of appropriate 
instructional materials based on organized course outlines. 
However, the level of use among agencies remains relatively 
low. There has been an accompanying increase in the 
organization of resource materials into resource libraries. 



Supporting Data. This key element is defined as using appropriate materials based 
on organized course outlines. In four of 11 agencies visited to collect interview data, 
instructors reported that their programs were using ESL course outlines which linked 
competency lists to specific levels, objectives, and sets of materials. In three of these four 
agencies, instructors were using a variety of books based on well-planned course outlines, 
and all reported that they were satisfied with these materials. 

In our other agencies visited, ESL instructors were attempting to use one textbook 
series for all program levels. Instructors reported that they were dissatisfied with the 
materials at all instructional levels in these agencies. This was particularly true in three of 
these four agencies, where the adopted textbook series had a strict life skills focus. 
Instructors complained that often such texts "lacked grammatical structure," and that they 
"needed to be supplemented too much with other materials." 
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At all four of the agencies where well-organized course outlines were both in place 
and in use, resource libraries also existed and were being utilized. In addition, two other 
agencies were in the process of establishing resource libraries. The establishment of 
resource libraries may be associated with the process of organizing course outlines based on 
student needs, and with the feeling that one textbook series is not likely to satisfy students 1 
needs at all program levels. 

The interpretations of the managers regarding the use of a variet of instructional 
materials paralleled those of instructors. Half of the managers interviewed across all 11 
sites reported that their ESL programs had seen major changes in the types of instructional 
materials in use since the advent of the state's CBAE mandate. 

For this area of CBAE implementation, the ISAM instrument defines the "use of 
appropriate instructional materials" as the presence or absence of course outlines that 
integrate life skill competencies and instructional Tfiaterials. A comparison of ISAM scores 
between 1983-84 (n=64 agencies) and 1984-85 (n=55 agencies) demonstrates positive - 
but not statistically significant - changes in the use of appropriate instructional materials. 
In 1984-85, 42% of managers reported the use of appropriate instructional materials in 
their agencies, while 39% of instructors sampled reported similar results. 

The CASAS Instructor Survey results support the finding that some changes have 
occurred in the choice in selection of instructional materials, with 69% (n=170) reporting 
moderate to significant change in choice, selection, and use of materials as a result of CASAS 
adoption. 

Interview data revealed further information on the impact of CBAE implementation on 
curriculum in local agencies. There was consensus in ten of 11 agencies visited that 
curriculum had become "more cohesive and organized" over the past two to three years, and 
that this was associated with the CBAE mandate. Thirteen out of 19 managers interviewed 
stated that organization of curriculum was a major program change over this period of time. 
To illustrate this finding, one instructor commented that she felt her program was now so 
well organized that, "the program could continue even if I leave." Considering the typically 
high turnover rate of adult education instructors in many agencies, this effect of CBAE on 
curriculum organization is important for future programming. 



8. Use a Variety of Instructional Strategies 

Managers and instructors believe that they are using a 
variety of instructional strategies, but students do not 
consistently perceive this use. ESL students often express 
a need for more opportunities to practice language, and ABE 
students often prefer more student-to-student communication. 



Supporting Data. ISAM results for 1984-85 indicate that a significant percentage 
of instructors and program managers perceive that a variety of instructional strategies are 
in use in their programs. Of the instructors sampled (n=56) in 1984-85, 82% reported use 
of a viariety of instructional strategies, while 76% if managers sampled (n=54) reported 
similar levels of use. The managers' sampled showed the greatest positive movement frc i 
1983-84 to 1984-85 (53% to 76%). 
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Based on interviews with 52 ESL students in 11 agencies visited, nine of 11 agencies 
were displaying some variety in instructional delivery in ESL classes, such as by dividing a 
class into two groups to address different levels of need. 

However, in only one of the 11 agencies did students feel that effective small 
group techniques were in use. ESL students consistently spoke of the need for more 
opportunities for individualized or small group conversation practice. Students often told 
interviewers that since, in their daily lives outside of school, they were "too shy" or 
Americans were "too busy for small talk," they therefore needed to practice conversation 
more in the classroom. When asked about their instructors' techniques, students always 
cited whole-group interaction and could seldom identify any alternative form of small group 
or individualized instruction taking place. According to one student, "instructor always says, 
'Why don't we talk more,' but won't be quiet so we can." 

Twelve ABE students interviewed in the 11 agencies reported that most class 
instruction was delivered in an individualized mode, and that they enjoyed it. Some, 
however, expressed a need to communicate more with other students and with the instructor. 
Two of 11 ABE instructors interviewed who were teaching in an individualized classroom 
setting described creative ways they used to add small-group and large-group class sessions 
focused on identified life skill needs of students. 



9. Develop, Implement, and Maintain 
Staff Development Programs 

The focus on CBAE implementation over the past three years 
has been associated with an overall increase in staff 
development opportunities, and in increased instructor 
participation in CBAE* related and other professional 
conferences at regional and state levels . Focused, 
coordinated staff development efforts at local program 
levels also have increased. The $2500 grant that is 
earmarked for CBAE implementation in the Section 306 
funding process is frequently identified as an important 
factor in the increase in staff development opportunities. 



Supporting Data 

Overall Increase in Staff Development Opportunities. A comparison of ISAM scores 
from 1983-84 to 1984-85 reveals that a positive, significant increase in the implementation 
of staff development activities has occurred. ISAM data for 1984-85 suggests that staff 
development programs are operational in 69% of the agencies sampled, representing a 22% 
increase from 1983-84 to 1984-85. 

In nine of 11 agencies visited to collect interview data, staff members reported, 
that there were staff development activities in place that were "helpful." Several 
instructors in these agencies reported that successful staff development activities helped to 
promote an overall sense of "professionalism" among the staff. 
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Fifteen of 19 managers interviewed reported increased staff participation inCBAE- 
reiated staff development activities. According to one manager, "Before the mandate, 
everyone was 'doing their own thing. 1 The CBAE staff development activities pulled the staff 
together and gave them more of a sense of program cohesiveness." 

Managers frequently associated the availability of an additional $2500 grant in 
Section 306 funds with the increase in staff development opportunities. One comment heard 
from several managers was that there was H no staff development" until the appearance of 
the CBAE mandate and the $2,500 grant augmentation. 

Patterns of Successful Staff Development. In the 11 agencies visited 
successful staff development appeared to include: administrative support, needs 
assessment, t ir ~* flexibility, and paid release time. Of these four key features, paid release 
time was cited least frequently as an important factor. When most of these features were in 
place, staff development programs were seen as successful. In contrast, when only some 
features were present, staff development appeared to be less successful. 

In one "high implementing" agency, for example, no paid release time was availableto 
instructors. However, by assessing instructors' staff development needs, and by offering 
staff development at several alternative times during the day, an effective program of staff 
development was carried out. On the other hand, in one "low implementing" agency where 
paid release time was available, there was no effective needs assessment and no time 
flexibility. The result was an ineffective staff development program. 

Remaining Problem Areas. Several remaining problem areas in the delivery of staff 
development at the local agency level emerged from the interview data. These included: lack 
of outreach to outlying sites and evening instructors, narrow interpretation of fundable staff 
development activities on the part of managers, lack of appropriate staff development for 
ABE instructors, and lack of involvement of aides in staff development. 

In all but the smallest agencies visited, managers had chosen a limited number of sites 
for initial CBAE implementation. Staff development was provided at these sites on a regular 
basis. Nine of 19 managers interviewed felt that their staff development efforts needed to 
be broadened to include more staff, particularly at outlying sites, and in evening programs. 

In some agencies, instructors commented that managers were only willing to fund 
instructor attendance at staff development activities clearly labeled as "CBAE." One 
instructor commented that this prevented her from attending conferences sponsored by 
professional organizations in her field, which in the past had been quite helpful to her. 

The 12 ABE instructors interviewed in the study reported that they are not well 
served by existing staff development activities. They do not generally participate in Section 
306-supported staff development activities together with ESL instructors because they feel 
thetwo groups have different needs. Several ABE instructors expressed a preference for 
being grouped with instructors from learning centers or high school diploma programs for 
staff development purposes. 



None of the five aides interviewed in the 11 agencies visited were included in staff 
development activities, although all of these aides performed key program functions related 
to placement of students or curriculum organization. One of the five was not interested in 
staff development, while the other four were uncertain if it would be "appropriate" for them 
to attend such activities. 



B. Other Findings of the Study 

This section presents findings that emerged beyond the framework of the nine key 
elements of CBAE discussed in the previous section. Some of these findings are supported 
primarily by single data sources, but, nonetheless, are valuable for understanding the 
patterns of CBAE implementation statewide. Other findings are derived from analyses of 
data collected in the interview sample of 11 agencies thai were examined in depth. 

For purposes of this study, the 11 agencies examined were categorized by relative 
level of CBAE implementation - "high, medium, and low" - as described in Chapter IV. The 
findings in this section are reported in terms of this categorization. The findings are 
reported in following three parts: 

1. Student Level Findings 

2. Classroom Level Findings 

3. Agency Level Findings 



1. Student Level Findings 



Achievement Gains 

After 100 hours of instruction, students from "high 

implementing" agencies achieved an average gain on the 

CASAS Survey Achievement Yest that was greater than 

that achieved by students from "low implementing" agencies. 

Moreover, when gains were broken down by program and 
level, the "high implementing" agencies demonstrated higher 

gains for every program and level than the "low 
implementing" agencies. 



Supporting Data. Analysis of CASAS SurveyAchievement Test statewide results 
for all programs and levels in 1983-84 and 1984-85 reveal a mean achievement gain of 4.2 
(1983-84) and 4.3 (1984-85) scaled scores, as indicated in Table 4. 
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TABLE 4 



Statewide Mean Achievement Gains, 1983-84 and 1984-85 







1983 - 


84 




1 984 - 85 






MEAN 


SD 


n* 


MEAN 


SD 


n* 


Pre-Test 


206.4 


14.2 


3950 


207.8 


15.3 


6413 


Scores 










Post-Test 


213.2 


15.6 


3950 


214.6 


15.7 


6413 


Scores 












Achievement 


6.8 


9.9 


3950 


6.8 


10.0 


6413 


Gain 


p<.001 








p<.001 



* (n refers to number of students) 



Additional analysis of scores for 1984-85 were conducted focusing upon a sample 
of agencies that were described earlier as "high implementing" and "low implementina" 
agencies. After 100 hours of instruction, students from "high implementing" agencies 
achieved an average gain on the CASAS Survey Achievement test greater than that 
achieved by students from "low implementing" agencies. Table 5 compares these mean 
gain scores. 



TABLE 5 



CASAS Survey Achievement Test Gains (1984-85) 
for "High Implementing" and "Low Implementing" Agencies 





MEAN 
GAIN 


SD 


n* 


■ % OF VALID 
SCORES 


High CBAE 

Implementing 

Agencies 


+4.93 


7.79 


3 


74% 


Low CBAE 

Implementing 

Agencies 


+3.53 


8.39 


3 


57% 



* (n refers to number of agencies-- no test of significance made) 
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A comparison of the mean gain scores for M high M and "low 11 implementing agencies indicates 
that in every analysis, M high implementing" agencies demonstrated higher gains than did "low 
implementing" agencies. Table 6 presents a comparison by program. 



TABLE 6 



Comparision of CASAS Survey Achievement Test Mean Gains by Program and 
Level for "High Implementing" and" Low Implementing" Agencies 





IMPLEMENTATION 
RATING 


MEAN 
GAIN 




STANDARD 
DEVIATION 


n* 


% OF VALID 
SCORES 


ESL 


High 

Implementing 
Low 

Implemented 


5.62 
4.49 


(n.s.) 


8.63 
9.86 


182 
183 


66% 

(p<.001) 

34% 


ABE 


High 

Implementing 


6.58 


(n/a) 


6.21 


13 


76% 

(n/a) 




Low 

Implementing 


N/A 




N/A 


N/A 


N/A 


Multi- 
Level 
(ABE) 


High 

Implementing 
Low 

Implementing 


3.99 
3.87 


(n.s.) 


9.53 
7.74 


17 
36 


94% 

(p<.01) 

57% 



* (n refers to number of students) 



The small number of ABE scores in Table 6 above makes interpretation of that aspect of the 
data difficult, except to note that the major source of variance between the "low" and "high" 
implementing at the program level would be attributable to ESL programs. 
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To capture this further, the following breakdown of mean gain scores by ESL level 
presented in Table 7 confirms that the direction of the difference in scores between "high" 
and "low" implementing agencies is consistent. 



TABLE 7 



Comparison of CASAS Survey Achievement Test Mean Gains by ESL Program 
Level for "High Implementing" and "Low Implementing" Agencies 





IMPLEMENTATION 


MEAN 


STANDARD 


n* 


% DP VAI ID 




RATING 

1 if * III »VI 


GAIN 


DPvIATinM 

UC V \r\ I \\J\ I 




OOUnto 


Beginning 


High 


6.39 


10.28 


73 


55% 


ESL 


Implementing 












Low 




(n.s.) 




(p<.001) 




5.13 


10.67 


79 


22% 




Implementing 










Intermediate 


High 


4.23 


6.71 


90 


74% 


ESL 


Implementing 












Low 




(n.s.) 




(P<-01) 




2.43 


9.47 


70 


50% 




Implementing 










Advance 


High 


9.21 


8.89 


19 


95% 


ESL 


Implementing 
















(p<.10) 




(n.s.) 




Low 


7.75 


5.09 


34 


85% 




Implementing 











* (n refers to number of students) 



Class levels for Table 7 were defined by CASAS pretest form. Both mean gain scores and 
percent of valid test scores are greater for "high implementing" than "low implementing" 
agencies. Scores are also more tightly clustered for the sample of "high implementing 
agencies" than" low implementing agencies", except at the advanced ESL level. 
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2. Classroom Level Findings 



a. Student Retention 

Although student retention rates statewide did not change 
significantly during the initial three-year period of CBAE 
implementation according to the ABE Class Questionnaire, 
one sampled "high CBAE implementing" agency had a 
significantly higher student retention rate than did a "low 
CBAE implementing" agency that was comparable in size 
and student population. In addition, CASAS data from a 
JDRP submission suggest that agencies having an 
assessment system which enables instructors to place, 
monitor, and certify student progress results in high 
levels of student retention. 

Supporting Data. Table 8 below presents statewide averages of total student 
enrollment reported as retained in Section 306 ABE and ESL classes. 



TABLE 8 



Average Retention Rates by Year 





1982-83 


1983-84 


1984-85 


Average Retention 


50% 


59% 


49% 


Rate Per Class 


(n=446)* 


(n=561)* 


(n=984)* 


* (n refers to number classes -- 


1982-85 p < 


.001) 





A comparison of retention rates reported for on large H high implementing- and one 
large "low implementing- agency in 1984-85 follows: 



TABLE 9 



1984-85 Average Retention Rate (%) by Level of Implementation 

(At 100 Hours of Instruction) 





RETENTION RATE 


NUMBER OF CLASSES 


High 


66.5% 


33 


Implementing 






Low 


47.4% 


43 


Implementing 







* ( P < .01 ) 



9 

ERJC 



33 38 



The average percent retention for the "high implementing" agency is well above the 
statewide average in 1984-85, while the average percent retention for {he "iow 
implementing" agency is below the state average. 

The direction of the difference between the "low implementing" and "high 
implementing" agencies remains consistent when compared by program level, and is 
especially evident in the beginning level of ESL, as Table 10 displays. 



TABLE 10 

Average Retention Rate (%) By ESL Level and Level of Implementation 

(At 100 Hours of Instruction) 





AVERAGE % 


NUMBER OF 
CLASSES 


Beginning 
ESL 


High 

Implementing 
Low 

Implementing 


72.8% 
42.4% 


5 

( P < .05 ) 

25 


Intermediate 
ESL 


High 

Implementing 
Low 

Implementing 


59.7% 
55.6% 


17 

( n.s. ) 

11 


Advanced 
ESL 


High 

Implementing 
Low 

Implementing 


59.7% 
51.5% 


5 

( n.s. ) 

3 



Table 11 on the next page presents data on student retention for the 1983-84 
program year from research submitted by CASAS to the federal Joint Dissemination Review 
Panel (JDRP). The table supports the claim that agencies implementing a competency-based 
assessment system exhibit higher rates of retention. Section A of Table 11 presents 
enrollment data prior to instruction, and Section B presents enrollment data following 100 
hours of instruction. 
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TABLE 11 



Comparisons of Average Student Retention of CASAS Consortium 
and Non-CASAS Consortium Classes 



A. PRIOR TO INSTRUCTION 




n* 


average enrollment 


CASAS Consortium 


14 


26.6 


Non-CASAS Consortium 


14 


29.8 


B. AFTER 100 HOURS OF INSTRUCTION Retention Rate 




n* 


average enrollment 


CASAS Consortium 


14 


10.8 40.5% 


Non-CASAS Consortium 


14 


5.7 19.1% 


* (n refers to number of classes - p < 


.05) 





The data presented above suggest that agencies which are members of the CASAS Consortium 
(and are, presumably, implementing a competency-based assessment system) are more 
likely to exhibit higher student retention rates than agencies which are not. 



b. Use of Aides 

Overall, use of aides in programs appears to have remained 
constant. Two data sources (interviews and CASAS Instructor 
Survey) suggest some change in the use of aides, in the 
direction of more non-instructional activities (performing such 
tasks as placing students, organizing materials, and organizing 
testing). However, a third data source (ABE Class 
Questionnaire) strongly suggests a pattern of littlo change 
over a three-year period. 



Supporting Data. Over a period of three years (1982-83: n=458, :983-84: 
n=561; and 1984-85: n«984), ABE Class Questionnaire results indicate that the average 
number of aide hours available per classroom have remained constant, at slightly over 3.5 
hours per week, per class statewide. Of those hours, two-thirds were consistently used for 
direct instructional assistance, and one-third for non-instructional assistance - such as 
curriculum organization, student record-keeping, and assessment - over the three-year 
period. 
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However, interview data indicated that aides were in use in four of 1 1 agencies 
visited. Of a total of five aides interviewed in the 11 agencies, primary duties were 
described as: placing students, organizing materials, and organizing testing. Three of the 
five aides described their current job responsibilities as a shift away from 
assisting with classroom instruction, the previous focus of their work. When asked how the 
change in job focus had come about, one aide responded, "It just happened. We had to do more 
paperwork because of a larger program at a new site." She also said that she missed working 
with students, and welcomed a recent change in her job that involved her more in placing 
students into the program. 

Furthermore, according to the CASAS Instructor Survey, 19% of the instructors 
reported moderate to significant change in the use of instructional aides as a result of 
implementing the CBAE elements, particularly through the CASAS program. Comments on the 
Survey indicated that this use of aides typically involved record-keeping functions and 
assessment. 

In summary, overall use of aides in programs appears to have remained constant. 
While interview data suggest a shift in the job responsibilities of aides from instructional to 
non-instructional activities, and while the CASAS Instructor Survey suggest a moderate shift 
in the use of aides from instruction to record-keeping, ABE Class Questionnaire survey data 
(with a substantial number of respondents, as noted above) do not support these patterns. 



c. Life Skills as the Curriculum Focus of CBAE 

A substantial percentage of instruction in adult basic education 
(ESL and ABE) classrooms statewide is devoted to life skills- 
oriented content. in interviews, managers defined CBAE in 
terms of the broad, program level implementation process, 
while instructors defined CBAE primarily in terms of life skills 
curriculum focus. ESL Instructors were split In their views of 
the appropriateness of life skills at different program levels, 
but many ABE instructors expressed the concern that a 
life skills focus, as they interpreted it, was not appropriate 
for their students. 



Supporting Data. The CBAE mandate in California emphasized an increased life 
skills curriculum focus. Findings on the impact of this curriculum focus fall into four major 
subcategories: (1) Overall Program Effects, (2) Differing Interpretations of CBAE, (3) 
Perceived Relevance of CBAE/Life Skills at Different Program Levels, and (4) Narrow 
Interpretations of Life Skills. 



Overall Program Effects. Statewide averages of percentage of life skills taught per 
class were reported on the ABE Class Questionnaire as noted in Table 12 on the next page. 
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TABLE 12 



Statewide Averages of Percentage of Life Skills Taught Per Class 





1982-83 


1983-84 


1984-85 


Average % of Classtime 


33.7% 


40.6% 


40.0% 


Devoted to Life skills 


(n=458) 


(n=561) 


(n=984) 


Per Class 





* ( 1982-85 p < .01 ) 



These data indicate that a life skills-oriented curriculum is seen as a major thrust of 
instruction in adult basic education classrooms throughout the state. 

Instructors in the 11 agencies visited frequently mentioned how lessons based on 
life skills accelerated communication in the classroom. As one ESL instructor noted, 
"Students in a grammar-based program can't ask life skill questhns - they don't have the 
language needed to communicate needs." A student in a different agency, which taught 
grammar-based ESL, said, "There are things to tell instructor, but can't tell her. Hard to ask 
questions - so much to ask, but no way." 

A comparison of average percent of class time devoted to life skills instruction in one 
large "high implementing" and one large "low implementing" agency in 1984-85 is shown in 
Table 13. 



TABLE 13 

1984-85 Average % of Classtime Devoted to Life Skills Instruction 
in One "High Implementing" and One "Low Implementing" Agency 





AVERAGE % 


NUMBER OF CLASSES 


High 


42.2% 


33 


Implementing 










(n.s) 


Low 


35.6% 


43 


Implementing 







Comparing the data in Tables 12 and 13, it becomes apparent that the average 
percentage of time devoted to life skills instruction is slightly above the state mean for the 
"high implementing" agency, while the average percentage for the "low implementing" agency 
is somewhat below the state average. 
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The direction of the difference between the "high implementing" and "low 
implementing" agency remains consistent when compared by program level, and is especially 
evident in the beginning level of ESL, as the following Table 14 displays. 



TABLE 14 



Percentage of Classtime Devoted to Life Skills Instruction by ESL Level in 
One "High Implementing" and One "Low Implementing" Agencies 





AVERAGE 


% 


NUMBER OF CLASSES 


Beginning 


High 


60.0% 




5 


ESL 


Implementing 














( P < -05) 






I 0\A/ 


O / .c, /o 




25 




Implementing 








Intermediate 


High 


37.9% 




17 


ESL 


Implementing 














(n.s.) 






Low 


33.2% 




11 




Implementing 








Advanced 


High 


45.8% 




5 


ESL 


Implementing 














(n.s.) 






Low 


31.7% 




3 




Implementing 









Differing Definitions of CBAE. All staff members interviewed from 11 agencies were 
asked to define "competency-based adult education" in their own words. These definitions 
tended to reflect either (a) the agency's stage of CBAE implementation, or (b) the role of the 
particular respondent (instructor or manager). . 

All but one agency had successfully implemented at least one "key element" of CBAE, 
and definitions within each agency tended to reflect those elements of CBAE that had already 
been implemented. For example, in an agency where substantial work had been done in 
assessment, the definition of CBAE that was given consistently emphasized "measurement of 
growth." In another agency where record-keeping on student competency attainment had 
been successfully implemented, CBAE was frequently defined as "documentation of student 
learning." 

To illustrate the impact of an individual's own role on CBAE definitions, managers 
typically saw CBAE as a "process" of some sort, while instructors consistently described it 
in terms of "life skills" curriculum content. In one manager's view - typical of many stated 
during interviews - "CBAE is addressing needs of students, planning instruction, and then 
assessing whether students have learned what is taught." 
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ESL instructors, however, consistently defined CBAE in terms of life skills content, 
without much emphasis on process. A characteristic definition offered by ESL instructors 
was: "teaching life skills that people need to survive." ABE instructors also focused on 
curriculum content, but their definitions more often focused on the "application of basic skills 
to life skills." 



Perceived Relevance of CBABLife Skills at Different Program Levels. There was 
variance among agency staff members interviewed as to the appropriateness of a life skills 
curriculum approach at all levels of adult basic education programming. 

Managers generally expressed confidence in the relevance of a CBAE approach at all 
program levels, frequently suggesting that it should be expanded beyond its current focus 
(ESL and ABE) to include other areas of instruction such as high school diploma programs. 

There was less unanimity on the part of ESL and ABE instructors. Of 38 ESL 
instructors interviewed, 14 said CBAE was appropriate at all levels of instruction, 12 said it 
was most appropriate at beginning levels, and eight saw it as most appropriate at advanced 
levels. Few problems were expressed about the intermediate levels. Opinions expressed by 
instructors were not generally tied to the levels in which the individual instructors taught. It 
is interesting to note, however, that in two agencies where ESL instructors taught at more 
than one level of instruction daily, no problems were encountered with teaching life skills at 
any level. 

When ESL instructors offered examples of the problems of CBAE instruction at 
specific levels, most examples focused on the advanced level. One instructor said that at 
this level, "Life skills are too simple, easy, or repetitive." Another cautioned that, "Life 
skills should be done selectively at advanced levels - a lot of their interests can't be seen on 
competency lists." Other comments were, "Materials are hard to find at this level," and, 
"Identifying life skills appropriate for advanced level students, together with accompanying 
basic skills, is a difficult task." 

When students were interviewed, however, they tended to express approval of the 
inclusion of a life skills focus in the curriculum. All ESL students interviewed, for example, 
could identify life skills that they had learned in class and were using in their daily lives. 
Whereas some instructors commented that students only needed to learn vocabulary words to 
be able to perform needed life skills, as an illustrative example, one student disagreed, 
saying that he seemed to be studying only vocabulary, and he needed to have more practice in 
using the vocabulary in realistic situations. 

Some ESL instructors commented that when students lived in a community where 
their native language was in wide use, it was unnecessary to teach them life skills. 
However, several students disagreed. One Hispanic woman who had been in the United States 
for 21 years *old an interviewer, "I have always been able to do things in Spanish, but 
learning how to do these same things in English has given me confidence and independence." 
She went on to say that she could choose any doctor now, not just one who spoke her 
language. Students frequently made similar comments, to the effect that they wanted to 
become "more independent." 
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ABE instructors frequently expressed the concern that a competency-based life 
skills approach, at least as defined by existing competency lists, was not appropriate for 
their students. They felt that many of the life skills on these lists did not relate to the 
students 1 lives. By contrast, however, all ABE students interviewed were able to identify 
life skills that they had learned or that they needed to learn. In an interesting juxtaposition 
of ABE instructor and student viewpoints on the appropriateness of a life skills curriculum 
focus, one instructor reported that she had added a unit on job skills into her curriculum only 
because management had told her to do so. Later, two students in her class stated that the 
unit on job skills was very interesting, and one said she had been able to "write a resume for 
my husband, and he got the job!" 

In summary, although there is some variance in the attitudes of ESL and ABE 
instructors toward the appropriateness of life skills as a curriculum focus, the majority of 
ESL and ABE students interviewed support such an approach. 



Narrow Interpretations of Life Skills. It was not uncommon to find that some 
instructors were interpreting the life skills focus of CBAE rather narrowly, and that some 
tended to separate life skills application activities from basic skill development activities. 

There was a tendency on the part of some ESL instructors, for example, to shift their 
entire instructional focus to a surface-level treatment of life skill applications, at the 
possible expense of not devoting sufficient attention to basic underlying communication skills 
that needed to be developed in order to enable students to perform specific life skills. 

Examples of the latter problem of separating life skills activities from basic skills 
activities were encountered in more than one program where ESL students attended a 
"grammar class for one hour, followed by a "life skills" class the next hour. 



d. Improved Communication and Program Planning 

In interviews most agency staff members reported "better 
communication" with each other, and most managers cited 
more involvement with staff and programs, as well as an 
increase in communication with other agencies. Examples of 
such interagency sharing included thu spontaneous formation 
of two separate interagency program coordinators' networks 
in different parts of the state. 



Supporting Data. All data presented in this section are based on interviews and 
are discussed in the following four categories: (I) Enhancement of Planning and 
Communication at the Agency Level, (2) The Conduct of Planning: Four Scenarios of "High 
Implementing* 9 Agencies, (3) Planning and Communication in "Low Implementing" Agencies, 
and (4) Enhancement of Interagency Linkages. 
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Overall Enhancement of Planning and Communication at the Agency Level. In most of 
the agencies visited, increased communication and sharing among staff members was 
consistently identified in interviews as an important program-level change. Thirteen of 19 
managers reported that the CBAE mandate provided a focus for managers, instructors, and 
counselors to meet and plan the CBAE implementation process. An overall effect of this 
mandated planning process seemed to be the establishment of communication channels that 
may not have existed before. Thus, the CBAE mandate appears to have provided local agency 
planning teams with both the incentive to meet, and an agenda to discuss when they met. 



The Conduct of Planning: Four Scenarios of "High Implementing" Agencies. Each of 
the four agencies identified as "high CBAE implementing" exhibited effective, but somewhat 
different planning and communication patterns. The communication patterns of these four 
agencies are described below. A common finding in all four agencies was that each of the 
approaches was consistently viewed positively by staff members, who felt "ownership" of 
the results. 

In all four agencies - although there were differences in specific approach - the 
following three key features of an effective planning process were consistently in evidence: 
(1) paid release time for staff members, (2) administrative support, and (3) some real 
impact on the direction of programming. In this area of planning, then, in contrast with the 
findings on staff development reported earlier in "key element" number nine, paid release 
time was consistently seen as necessary. 

At the first agency, a "team" approach was clearly in place. Planning consisted of a 
formally structured tim3 every four to six weeks. The members of the team received paid 
release time and h? a voice in the planning process with the manager. The team was told, 
"Look at what is involved in implementing CBAE, decide if ft is beneficial, and if you decide 
it's not, well withdraw from Section 306 funding." This made the staff feel that they truly 
had administrative support for the decisions they wouid make, and when they decided to 
become involved in CBAE, they did not take the decision lightly. As the manager noted in an 
interview, "If staff have input and ownership, they feel better about change." Through the 
planning process that evolved in this agency, the instructional staff arrived at a number of 
constructive curriculum innovations, as well as a restructuring of the student levels in their 
program. 

A combination of planning and staff development worked well in the second "high 
implementing" agency. Staff members were actively involved in planning, and their staff 
development needs were assessed. All five instructors interviewed at this agency mentioned 
that needs assessment for staff development took place, and they all characterized it as 
"helpful." The delivery of staff development activities involved both outside consultants and 
persons from with the program, and both sources of knowledge were appreciated. Planning 
or staff development meetings always had small-group tasks focused on an assessed common 
interest. This provided time for instructor "sharing." The manager in this program had the 
strong and clearly-stated expectation that staff members would attend these activities, and 
the manager set the example by always attending. Instructional staff received compensation 
for planning and staff development time, and this process was described by one staff member 
as "very worthwhile and beneficial." 
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The third "high implementing" agency appeared to have generated a sense of "going 
beyond formal planning." At the start of the CBAE implementation process in this small 
agency, tightly structured regular planning meetings were held to address the requirements 
of the state's mandate. Staff members were involved in the decision-making process from 
the outset. However, as the new CBAE program approach took hold, staff members felt less 
need to meet formally, and they began to move to informal sharing on almost a daily basis. 
Although formal planning meetings are now held much less frequently than at the start of 
CBAE implementation, periodic meetings are still held, and staff members are paid for 
attendance. 

In the fourth "high implementing" agency a nucleus of key people undertook planning 
for CBAE implementation. A large agency with multiple sites, members of this agency's 
planning team were drawn from al! the agency's major sites. Planning group participants 
tended to hold contract positions, and they were viewed as leaders by staff members at their 
sites. As a part of the planning process, staff members at all sites were regulariy asked » 
both formally and informally to state their needs and ideas, which then could be considered 
by the planning team. This communication and planning process appears to have continued 
after the initial emphasis on CBAE implementation. Communication in this agency also is 
enhanced by the monthly production and wide Distribution of a staff newsletter containing 
curricular information. An exemplary staff development program provides additional 
channels for staff communication in this agency. 

Planning and Communication in "Medium" and "Low Implementing "Agencies. In 
"medium" and "low" implementing agencies, planning for CBAE implementation was much less 
in evidence than in the "high implementing" agencies previously described. There appeared to 
be less emphasis on staff involvement, and less commitment to a collaborative problem- 
solving process. 

In four of the seven "medium" and "low" implementing agencies staff members spoke 
of planning on an "as needed basis," and interviewers identified little evidence of organized 
planning processes in these agencies. 

In three of the four "low implementing" agencies, instructors voiced the desire to 
have more planning meetings. This was in spite of the fact that the majoriW of these 
instructors were part-time and probably could not be compensated for attending such 
meetings. Managers in these agencies stated that they were reluctant to include staff in 
planning processes because they could not afford to compensate them. This finding suggests 
that the desire to have a real impact on the direction of programming may be at least as 
important a motivator for part-time staff members as paid release time. 

Enhancement of Interagency Linkages. Staff members in five agencies from the 11 
agency sample noted an increase in communication with other agencies that was associated 
with CBAE implementation. Several people said that they felt "more connected and less 
isolated" than before. Such interagency sharing also resulted in the independent formation of 
two separate interagency program coordinators* networks one in the southern part of the 
San Francisco Bay Area and the other in the east Los Angeles County area. 
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3. Agency Level Changes 



a. Varying Sequences of CBAE Implementation 

Many programs in the interview sample had some, but not 
all, elements of a CBAE system in place. There was also 
substantial variation in those elements that were first 
implemented by individual agencies. While the State 
Department of Education's CBAE mandate assumed a linear 
order of implementation beginning with a philosophy 
statement and competency list, many programs 
demonstrated more of a "nonlinear" order of 
implementation, emphasizing elements such as 
curriculum or assessment, depending upon perceived local 
needs. There was also a tendency within agencies to 
implement CBAE in larger, centralized sites with 
daytime program staff members. There was much less 
involvement of satellite sites and night time program 
staff members. 



Supporting Data.. Most staff members interviewed in the11 agencies felt that 
substantial changes had taken place in their programs during the first three-year period of 
the CBAE mandate. Of the 19 managers interviewed, 14 attributed these changes directly to 
the CBAE mandate, calling them predominantly "positive." 

Thero was, however, substantial variation in the first elements of a CBAE program, 
and in the order of other program elements, that were implemented by individual agencies. 
Some agencies tended first to address specific program areas where change was most 
clearly needed, such as curriculum or student placement, rather than follow the state's 
suggested order of implementation. 

In nine of 11 agencies visited, staff members had devoted substantial effort to 
implementing at least one of the key elements of CBAE. In each case, they were well 
satisfied with the work they had done, and the success of this initial work on one key element 
often appeared to motivate the development of additional CBAE program components. In one 
agency, for example, the first step was the development of a placement process through a 
series of staff planning meetings. After this placement process was field tested and 
implemented, instructors then found the need to continue communicating more closely about 
program concerns, and they began to examine what actually was being taught at each of the 
levels in the program. As a result, they began to develop course outlines. 

In the majority of agencies visited, managers chose to implement CBAE at only one 
site, and to involve a nucleus of people at that particular site in planning the CBAE 
implementation. Staff members comprising this "nucleus" were typically daytime 
instructors who taught more hours than other instructors. There tended to be much less 
involvement of instructors at "satellite" sites located away from the central site, and even 
less involvement of night time instructors. Also, managers and coordinators typically 
professed little knowledge of the activities of their night time instructors. For example, a 
resource instructor noted how she had extended a casual invitation to a night instructor to 
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come to a planning meeting, although she didn't usually do so because she believed that "all 
the night instructors work in the daytime/ To her surprise, the night instructor was 
excited to be invited, and attended the meeting. 

b. Responsiveness to Student Needs 

In the interview sample, managers reported increased 
assessment of students needs, and consequently "more 
responsiveness" to those needs. Instructors, however, 
reported that at the classroom level, assessing students' 
needs was problematic. 



Supporting Data. Thirteen of 19 managers in the interview sample described their 
programs as being more responsive to student needs, arguing that improved placement 
combined with student needs assessment procedures had made their programs "more 
relevant" to students. 

However, the majority of instructors reported assessing student needs less 
frequently than managers reported, and they seldom could identify curriculum changes that 
had come about as a result of such needs assessment. Instructors appeared to believe that 
student needs assessment was important, but they seemed less willing or able to change 
their teaching based on expressed student needs. 

The majority of students interviewed were unable to identify that formal needs 
assessment was taking place in their classrooms. More frequently, students said that they 
could ask instructors to teach them something they needed to know, and the instructor would 
help them. A few students, however, were not sure if such requests were appropriate for 
them to make. 

c. "Key Communicator" Role 

In "high implementing" agencies there has consistently been 
one individual a "key communicator" who has been most 
clearly responsible for facilitating CBAE program change. 
Based on the interview sample, the person in this role must 
have some authority to make change, must be paid to work 
sufficient non-instructional hours to bring about the desired 
change, and must be able to communicate effectively with both 
administration and staff. The "key communicator" has various 
titles (such as, "coordinator," "resource instructor," or 
"department chair") and generally works with a team of 
instructors and other staff members. 



Supporting Data 

The Importance of the "Key Communicator". One staff role consistently emerged at 
the agency level as a part of the CBAE implementation process: the "key communicator." 
The person in this role typically had to handle critical responsibilities as a leader in planning, 
coordinating, communicating, and generally "holding together" the implementation process in 
a coherent way. 
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At all four "high implementing" agencies, managers said that it was essential to have 
a person who clearly understands the process in order to implement CBAE. This was 
generally the case too, in "medium implementing" agencies. In interviews with managers 
across all 11 agencies visited, seven of 19 managers commented that CBAE implementation 
required one key staff member who was "sold" on the idea. As one manager noted, it 
requires, "one key person to communicate what the intent and benefits of the mandate are." 
Another manager said he learned as much as he could about the mandate, shared ideas with 
his staff, and then "waited for leaders to emerge." One of those leaders who emerged was 
such an effective communicator that she was selected to fill the role of coordinator. 

Patterns of the "Key Communicator Role. The following patterns emerged to 
describe the nature and operational context of the "key communicator" role: 

(1) The person in the role was a woman at all 11 agencies, except at one 
agency where there was a male/female team. 

(2) The person typically was paid as a "instructor on special assignment," 
and did not have management status or pay. 

(3) Those persons who were most effective in the position enjoyed strong 
management support, and exercised a free hand in areas that might 
normally be considered prerogatives of a manager. 

(4) Successful CBAE implementation did not occur in agencies where a 
'key communicator" was not operating effectively. 

Based on an analysis of interview data on successful "key communicators" in "high 
implementing" and "medium implementing" agencies, the follow conditions were identified as 
possible indicators of success: 

(1) The "key communicator" is most effective in a full-time position. 

(2) The "key communicator" must have some authority to make changes. 

(3) The "key communicator" must be a skilled program organizer. 

(4) The "key communicator" must be able to communicate successfully 
with administration and with staff 



Where these four conditions are not possible, CBAE implementation will probably be 
less than successful, as the following examples illustrate. 

Varying "Key Communicator" Scenarios. In one agency, the "key communicator" 
was making a real effort to organize the curriculum, and her efforts were appreciated by the 
instructional staff. However, she was paid only a few extra hours each week over and above 
her normal classroom teaching load to perform a coordination function, and she had no real 
authority to make changes. CBAE implementation in her program was spotty, and she could 
not get the support she wanted to make needed changes. 
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In another agency, the H key communicator position was full-time, filled by a 
capable team whose members each had separate strengths in communication and organization 
that complemented each other. However, they had limited authority to make real changes in 
the program and many of their efforts were ineffective. 

In a third agency, the "key communicator position was paid, with the authority to 
make program decisions, and the person in the position had sound organizational skills. 
However, while there was a supportive relationship with administration, communication with 
instructional staff was ineffective. 

The experience of a fourth agency serves further to illustrate the importance of the 
"key communicator position. In this agency a person held the position, for a time, who was 
unable to communicate effectively with either instructors or the manager. As a result, CBAE 
implementation was at a standstill. After she left, a new person was hired who 
communicated successfully with staff and administration, and within six months the program 
was making rapid progress toward CBAE implementation. 

In summary, in every agency that was effectively implementing CBAE, a "key 
communicator position was clearly established and filled by a motivated and efficient person 
who was appreciated by both management and instructional staff. 



d. Importance of Management Support 

"High Implementing" agencies consistently demonstrate 
strong management policy support for CBAE implementation. 
Without such support, successful implementation is 
difficult. 



Supporting Data. For successful CBAE implementation to take place, there is 
the clear need for commitment from managers. This commitment needs to take the form of 
financial as well as emotional resources. As one manager said, "Money and recognition are 
important, but recognition alone isn't enough. 

Of the 19 managers interviewed, 13 felt that their own presence was a key to 
successful staff planning activities, and eight felt that paid time for planning was important. 
According to one instructor, "The presence of management support makes you feel 
appreciated and validates what is being done, or encourages change to do things better." And 
one manager noted, "Move staff toward change only when there's time and resources. 
Otherwise you only create frustration," 

In "high implementing" agencies there appeared to be two common patterns of 
management support. In the first pattern, the manager left the implementation process in the 
hands of capable staff members, while offering praise, encouragement, and financial support. 
As one manager of a large agency who did this said, "Find the right people . give them the job, 
and then have sense enough to let them alone," In the second pattern, the manager developed 
expertise in CBAE, and took a direct hand in guiding implementation. In either case - 
whether indirectly or directly - the manager needs to convey tangible support of the CBAE 
implementation process for it to be successful. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 



This chapter presents overall conclusions and recommendations for further action at 
the state, local agency, and classroom levels that may be drawn from the findings reported 
in the previous chapter. The chapter is divided into the following six sections: 



A. State-Level Supports for Change 

B. Local Agency-Level Supports for Change 

C. CBAE Curriculum 

D. Assessment, Record-Keeping, and Guidance 

E. Instructional Strategies and Materials 

F. Staff Development 



Each section contains study conclusions followed by recommendations that emerge from tl a 
conclusions. 



A. State-Level Supports for Change 



Conclusions 



Focusing of 310 Funding. The state-level decision to concentrate Section 310 funds in 
four key content areas (curriculum.assessment, dissemination, and staff development) as a 
statewide technical assistance support structure appeared to contribute substantially to 
local agency-level implementation of a CBAE program framework. 

Mandate as Leverage. The state-level policy of mandating a CBAE program approach 
as a condition of Section 306 funding at the local agency level appeared to "leverage" change 
in areas of local agency programming that extended beyond the boundaries of programs and 
operations covered by Section 306 regulations. The institution of an additional $2,500 
"curriculum improvement grant" as a part of each 306 grant appeared to aid the 
implementation process. CBAE implementation became, in many agencies, an issue that 
extended beyond minimal compliance with state guidelines. 
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Program Improvement and Program Accountability. There were two underlying state 
policy goals associated with the CBAE mandate: 1) program improvement and 2) program 
accountability based on the establishment and maintenance of a uniform data base increased 
local agency capacity to document student growth. Of the two goals, the first - program 
improvement which mean* implementation at the local agency level of the key elements of 
CBAE contained in this report appeared to be more effectively addressed by local agencies. 
In the area of program accountability, while there was substantial technical progress 
(particularly through implementation of the CASAS system) in documenting student growth, 
there remained a need for further local use of newly-available data on student growth for 
purposes of program planning or program advocacy. 



Improvement of Interagency Communication. The CBAE mandate is strongly associated 
with improvement in frequency and quality of communication and sharing of information 
between local agencies. Regional and statewide consortia and networks established as a part 
of the Section 310 CBAE technical assistance support structure appear to have contributed to 
this outcome. 



Recommendations 



Focusing of 310 Funding. The practice of focusing 310 funding should be considered 
again in the future if state-level policy initiatives similar to the CBAE mandate are 
contemplated. 

Needed State-Level Technical Assistance for CBAE. There is a need xo determine the 
level and duration of ongoing state-supported technical assistance required for the 
maintenance and expansion of a mandated statewide innovation such as CBAE in California. 

Program Improvement and Program Accountability. There needs to be continued 
state-level policy emphasis on program improvement, with particular focus at the classroom 
level. There also needs to be added emphasis both on collecting program accountability data 
and on making use of student impact data at the program ievel to enhance program 
accountability and program advocacy. 

Specified 306 Budget Categories. Designation of specified incentive categories in the 
306 budget (such as the $2,500 curriculum improvement grant) should be considered in the 
future to support CBAE implementation or other priority service areas should the need arise. 
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B. Local Agency-Level Supports for Change 



Conclusions 



Strong Administrative Support Strong support from local agency administration 
appeared to contribute to successful CBAE implementation. Effective administrative support 
could take the form of either: 1) generalized policy support, or 2) direct involvement in 
guiding the steps of CBAE implementation. 

Emergence of the "Key Communicator". Successful CBAE implementation at the local 
agency level was strongly associated with the presence of an effective "key communicator," 
a coordinator or resource teacher who was usually without administrative rank, and who was 
charged with direct responsibility for coordinating CBAE implementation efforts. 

Formation of Agency Teams. The use of a collaborative team approach was associated 
with successful CBAE implementation. Local agency teams that included administrators, 
coordinators, guidance staff, and instructors were found to make major contributions in 
planning, curriculum development, assessment, and staff development. 

More Impact at Program than at Classroom Level. The initial thrust of CBAE 
implementation appeared to have its greatest impact at the agency or program level, rather 
than at the classroom level. The most clearly articulated aspects of the CBAE framework 
recommended by the state had to do with program level functions (such as competency 
identification, assessment, and staff development), and the focus of state-funded staff 
development and dissemination efforts tended to be on programs rather than classrooms. 



Recommendations 



Support for Agency-Level Use of Data. Although there is increased reporting of 
student impact data at the state level, there is still a need to help administrators and 
coordinators at the agency level to interpret and make practical use of student impact data 
for purposes of program evaluation and program advocacy. 

Support for Key Communicators. Since an effective key communicator at the agency 
level appears to be a critical element of successful CBAE implementation, it is important to 
provide continuing training and support to those individuals who fill this important function. 

Greater Focus on the Classroom Level. There needs to be added focus placed on the 
definition of, support for, and delivery of competency-based instruction at the classroom 
level. 
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C. CBAE Curriculum 



Conclusions 



Improved Curriculum Organization. The implementation of CBAE at the local agency 
level was associated with an overall improvement in the organization of ESL and ABE 
curricula. Improved curriculum organization took two characteristic and complementary 
forms: 1) the development or adaptation of competency lists and 2) the specification of 
course outlines that designated competencies by basic skill difficulty level and recommended 
teaching materials. 



Differing Understandings of CBAE. Local agency staff members exhibited two 
differing, characteristic Interpretations of the meaning of a CBAE program approach: 1) 
administrators and coordinators generally understood CBAE as a program process including 
the generic steps specified in the state framework while 2) teachers typically understood 
CBAE as "teaching life skills or survival skills.- 

Relevance of Life Skills to Students and Student Needs Assessment. All students 
interviewed could identify some specific life skills that they had acquired and used as a result 
of CBAE instruction. Most supported a life skills curriculum orientation. Student needs 
assessment apparently takes place in many programs, but identifiable program or curriculum 
changes made as a result of such needs assessment are often difficult to isolate. 

Appropriateness of Life Skills in Different Programs and Levels. ESL teachers believed 
a life skills curriculum focus to be most appropriate for beginning and intermediate level 
students. They suggested that such a focus was harder to maintain for advanced level 
students because students' needs and interests become more divergent at that level. ABE 
teachers reported some difficulty with integrating a life skills focus into their teaching. 



Recommendations 



Teacher Understanding of CBAE Processes. Teachers need to gain a better 
understanding of the generic process elements of a CBAE program as thay affect classroom 
instruction. They need to understand what, in addition to "teaching life skills," a CBAE 
approach involves. 

Improved Student Needs Assessment. Teachers need to be provided with additional 
simple, practical ways to assess student needs at the classroom level. They need to be 
trained to make changes in instruction based on needs assessment data. 

Life Skiiis Curriculum Focus. Teachers need continued training in the integration of 
basic skills and life skills. There is also a need to define more clearly those life skill areas 
that are appropriate for advanced level students in ESL and ABE classes. 
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D. Assessment, Record-Keeping, and Guidance 



Conclusions 



Achievement Gains. CASAS survey achievement test scores indicated that ESL and 
ABE students throughout the state at all program levels had made consistent and substantial 
gains in reading and listening comprehension skills that are related to life skills performance 
demands. Higher gains are associated with a higher degree of CBAE implementation at the 
local agency level. Although there was no change in the CBAE mandate, local agencies 
voluntarily choose to substantially increase their use of state-provided tests. There has also 
been increased accuracy at the state level in the ability to monitc student progress. 

Improved Quality of Assessment Local agencies report improvement in the following 
key areas of student assessment: 1) placement, 2) monitoring progress, 3) movement 
through programs, and 4) exiting programs. Improvement has been particularly noticeable 
in the area of student placement. 

Record-Keeping. Teachers consistently reported that keeping records to document 
individual student attainment of specific competencies was their greatest barrier to 
comprehensive CBAE implementation. No single successful approach to record-keeping 
emerged that could accommodate the reality of large, open entry/open exit classes with 
shifting student populations. 

Delivery of Guidance Services. Few local agencies appeared to have adequate 
resources !o assess and address the guidance needs of ESL and ABE students. Where guidance 
services were available, their characteristic focus was on high school diploma counseling for 
ABE students. Areas of largely unmet guidance need for ESL and ABE students included: 1) 
orientation to the program, 2) counseling for movement through the program, and 3) 
counseling for change in program or further education. 



Recommendations 



Teacher Training in Uses of Assessment. Continued training is needed at the local 
agency level in the uses of CASAS and other assessment processes for the purpose of 
placement, diagnosis, instructional materials selection, monitoring, and certification. 

Agency Support for Assessment. Agencies need to adopt and adapt existing processes 
or develop processes to assist teachers with scoring tests and returning results in a timely 
and understandable fashion. 

Development of Record-Keeping Processes. Agencies need to define clearly what kind 
of records teachers need to keep on individual student progress and competency attainment. 
Agencies also need to devise reasonable record-keeping methods that do not intrude 
excessively on the teachers 1 primary function of delivering instruction. Uses of feasible 
computerized management systems also need to be explored. 
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Improvement of Guidance Services. There is a need to improve student access to 
guidance support, particularly in the areas of orientation to program, movement through 
program, and change in program. Alternative means of delivering guidance services that do 
not rely on certificated guidance personnel - including but not limited to "teachers as 
counselors" - need to be explored further. 



E. Instructional Strategies and Materials 



Conclusions 



Instructional Materials. Teachers reported an increase in the variety of instructional 
materials in use in the classroom. This increase was associated with CBAE implementation. 
In "high-implementing" CBAE programs, course outlines that listed a selection of appropriate 
life skills-oriented materials by competency and basic skill level were found to be in use. 

Instructional Strategies. ESL and ABE teachers consistently reported the use of a 
variety of instructional strategies and grouping techniques. Student interviews did not 
always support this view, ESL students typically reported being taught through whole-group 
methods, and expressed the desire for small-group alternatives, ABE students typically 
reported being taught .through individualized methods, and also expressed the desire for 
small- group alternatives. 



Recommendations 



Expansion of Resource Libraries. Establishment or expansion of resource libraries 
containing appropriate basic skill and life skill materials at the agency and site level should 
be encouraged in order to enhance availability of appropriate materials. 

Training in Grouping Strategies for Teachers. Teachers should be trained in the use of 
alternative grouping strategies for delivering Instruction. Both ESL and ABE teachers need 
training in the use of small group formats such as pairs or teams. 
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F. Staff Development 



Conclusions 



Increased Frequency and Access. Teachers and administrators reported increased 
frequency of staff development activities - and access to these activities - at tha agency, 
regional, and state level. These changes were associated with CBAE implementation at the 
agency and state levels. The primary beneficiaries of staff development at the state and 
regional level appeared to be CBAE implementation team members, key communicators, and 
administrators* At the agency level teachers in daytime programs, particularly those located 
at central sites, appeared to have more consistent access to staff development than did those 
teachers in the night programs, those at outlying sites, or instructional aides. 

Factors for Success. Successful staff development at the local agency level 
consistently exhibited the following characteristics: 1) needs assessments of teachers were 
conducted, and staff development activities were arranged to address specific emerging 
needs; 2) staff development activities were offered more than once at alternative times to 
accommodate the different schedules of teachers; and 3) teachers were released from class 
or paid to attend staff development activities. Of these three factors, the third, while 
desirable, was found to be less important to successful staff development than the first two. 



Recommendations 



Expansion at the Local Agency Level There is a need to maximize access to staff 
development for classroom teachers in local agencies. There is also a particular need to 
provide staff development support to ABE teachers, instructional aides, and volunteers. 

Expansion to Outlying Sites and Evening Programs. There is a need at the local agency 
level to expand access to staff development so that teachers in outlying sites and evening 
programs may take advantage of staff development opportunities. 
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